🐟Fish Classification🔎

🐠(Prediction at the end)🔮

A Large-Scale Dataset for Segmentation and Classification

Authors: O. Ulucan, D. Karakaya, M. Turkan Department of Electrical and Electronics Engineering, Izmir University of Economics, Izmir, Turkey Corresponding author: M. Turkan Contact Information: mehmet.turkan@ieu.edu.tr

Paper : A Large-Scale Dataset for Fish Segmentation and Classification General Introduction

This dataset contains 9 different seafood types collected from a supermarket in Izmir, Turkey for a university-industry collaboration project at Izmir University of Economics, and this work was published in ASYU 2020. The dataset includes gilt head bream, red sea bream, sea bass, red mullet, horse mackerel, black sea sprat, striped red mullet, trout, shrimp image samples.

Purpose of the work

This dataset was collected in order to carry out segmentation, feature extraction, and classification tasks and compare the common segmentation, feature extraction, and classification algorithms (Semantic Segmentation, Convolutional Neural Networks, Bag of Features). All of the experiment results prove the usability of our dataset for purposes mentioned above.

Data Gathering Equipment and Data Augmentation

Images were collected via 2 different cameras, Kodak Easyshare Z650 and Samsung ST60. Therefore, the resolution of the images are 2832 x 2128, 1024 x 768, respectively.

Before the segmentation, feature extraction, and classification process, the dataset was resized to 590 x 445 by preserving the aspect ratio. After resizing the images, all labels in the dataset were augmented (by flipping and rotating).

At the end of the augmentation process, the number of total images for each class became 2000; 1000 for the RGB fish images and 1000 for their pair-wise ground truth labels.

Description of the dataset

The dataset contains 9 different seafood types. For each class, there are 1000 augmented images and their pair-wise augmented ground truths. Each class can be found in the "Fish_Dataset" file with their ground truth labels. All images for each class are ordered from "00000.png" to "01000.png".

For example, if you want to access the ground truth images of the shrimp in the dataset, the order should be followed is "Fish->Shrimp->Shrimp GT".

import numpy as np
import pandas as pd
import os
from pathlib import Path

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split

from sklearn.metrics import f1_score, accuracy_score, confusion_matrix, classification_report, roc_curve
from sklearn.model_selection import learning_curve, cross_val_score, GridSearchCV
from sklearn.model_selection import train_test_split

from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import RobustScaler,StandardScaler,MinMaxScaler

import warnings
warnings.filterwarnings('ignore')

image_dir = Path('../input/a-large-scale-fish-dataset/Fish_Dataset/Fish_Dataset')

# Get filepaths and labels
filepaths = list(image_dir.glob(r'**/*.png'))
labels = list(map(lambda x: os.path.split(os.path.split(x)[0])[1], filepaths))

filepaths = pd.Series(filepaths, name='Filepaths').astype(str)
labels = pd.Series(labels, name='Labels')

# Concatenate filepaths and labels
image_df = pd.concat([filepaths, labels], axis=1)

# Drop GT images
image_df = image_df[image_df['Labels'].apply(lambda x: x[-2:] != 'GT')]

# Shuffle the DataFrame and reset index
image_df = image_df.sample(frac=1).reset_index(drop = True)

# Show the result
image_df.head()

	Filepaths	Labels
0	../input/a-large-scale-fish-dataset/Fish_Datas...	Red Mullet
1	../input/a-large-scale-fish-dataset/Fish_Datas...	Trout
2	../input/a-large-scale-fish-dataset/Fish_Datas...	Sea Bass
3	../input/a-large-scale-fish-dataset/Fish_Datas...	Red Sea Bream
4	../input/a-large-scale-fish-dataset/Fish_Datas...	Trout

#Displaying a subsample of the dataset
fig, axes = plt.subplots(nrows=3, ncols=5, figsize=(15,10), subplot_kw={'xticks':[], 'yticks':[]})
for i, ax in enumerate(axes.flat):
    ax.imshow(plt.imread(image_df.Filepaths[i]))
    ax.set_title(image_df.Labels[i])
    
plt.show()

create_report(image_df)

DataPrep Report Overview

Variables ≡

Filepaths Labels

Missing Values

Overview

Dataset Statistics

Number of Variables	2
Number of Rows	9000
Missing Cells	0
Missing Cells (%)	0.0%
Duplicate Rows	0
Duplicate Rows (%)	0.0%
Total Size in Memory	1.9 MB
Average Row Size in Memory	222.0 B
Variable Types	Categorical: 2

Dataset Insights

Filepaths has a high cardinality: 9000 distinct values	High Cardinality
Filepaths has all distinct values	Unique

Variables

Filepaths

categorical

Show Details

Approximate Distinct Count	9000
Approximate Unique (%)	100.0%
Missing	0
Missing (%)	0.0%
Memory Size	1.4 MB

Stats Pie Chart Word Cloud Word Frequency Word Length

Length

Mean	96.3333
Standard Deviation	8.6415
Median	99
Minimum	83
Maximum	109

Sample

1st row	../input/a-large-s...
2nd row	../input/a-large-s...
3rd row	../input/a-large-s...
4th row	../input/a-large-s...
5th row	../input/a-large-s...

Letter

Count	656000
Lowercase Letter	580000
Space Separator	20000
Uppercase Letter	76000
Dash Punctuation	38000
Decimal Number	45000

• Filepaths contains many words: 7015 words

• The largest value (sea) is over 2.0 times larger than the second largest value (red)

Labels

categorical

Show Details

Approximate Distinct Count	9
Approximate Unique (%)	0.1%
Missing	0
Missing (%)	0.0%
Memory Size	673.8 KB

Stats Pie Chart Word Cloud Word Frequency Word Length

Length

Mean	11.6667
Standard Deviation	4.3207
Median	13
Minimum	5
Maximum	18

Sample

1st row	Red Mullet
2nd row	Trout
3rd row	Sea Bass
4th row	Red Sea Bream
5th row	Trout

Letter

Count	94000
Lowercase Letter	74000
Space Separator	10000
Uppercase Letter	20000
Dash Punctuation	1000
Decimal Number	0

Missing Values

Bar Chart Spectrum Heat Map Dendogram

Report generated with DataPrep

Modelling

Creating the model

import tensorflow as tf

# Separate in train and test data
train_df, test_df = train_test_split(image_df, train_size=0.85, shuffle=True, random_state=1)

train_generator = tf.keras.preprocessing.image.ImageDataGenerator(
    preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input,
    validation_split=0.15
)

test_generator = tf.keras.preprocessing.image.ImageDataGenerator(
    preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input
)

train_images = train_generator.flow_from_dataframe(
    dataframe=train_df,
    x_col='Filepaths',
    y_col='Labels',
    target_size=(224, 224),
    color_mode='rgb',
    class_mode='categorical',
    batch_size=64,
    shuffle=True,
    seed=42,
    subset='training'
)

Found 6503 validated image filenames belonging to 9 classes.

val_images = train_generator.flow_from_dataframe(
    dataframe=train_df,
    x_col='Filepaths',
    y_col='Labels',
    target_size=(224, 224),
    color_mode='rgb',
    class_mode='categorical',
    batch_size=64,
    shuffle=True,
    seed=42,
    subset='validation'
)

Found 1147 validated image filenames belonging to 9 classes.

test_images = test_generator.flow_from_dataframe(
    dataframe=test_df,
    x_col='Filepaths',
    y_col='Labels',
    target_size=(224, 224),
    color_mode='rgb',
    class_mode='categorical',
    batch_size=64,
    shuffle=False
)

Found 1350 validated image filenames belonging to 9 classes.

from keras.models import Sequential, Model
from keras.layers import Dense, Flatten, Dropout, GlobalAveragePooling2D

input_shape = (224, 224, 3)
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=input_shape ),
    tf.keras.layers.MaxPool2D(pool_size = (2,2)),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPool2D(pool_size = (2,2)),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPool2D(pool_size = (2,2)),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPool2D(pool_size = (2,2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(9, activation='softmax')
])

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 222, 222, 64)      1792      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 111, 111, 64)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 109, 109, 64)      36928     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 54, 54, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 52, 52, 32)        18464     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 26, 26, 32)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 24, 24, 32)        9248      
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 12, 12, 32)        0         
_________________________________________________________________
flatten (Flatten)            (None, 4608)              0         
_________________________________________________________________
dense (Dense)                (None, 512)               2359808   
_________________________________________________________________
dense_1 (Dense)              (None, 128)               65664     
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 128)               16512     
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 9)                 1161      
=================================================================
Total params: 2,509,577
Trainable params: 2,509,577
Non-trainable params: 0
_________________________________________________________________

from tensorflow.keras.optimizers import Adam

model.compile(optimizer="adam", loss='categorical_crossentropy', metrics=["accuracy"])

callback = tf.keras.callbacks.EarlyStopping(monitor='accuracy', patience=4)

history = model.fit(train_images, validation_data=val_images, epochs=6, callbacks=callback)

Epoch 1/6
102/102 [==============================] - 177s 2s/step - loss: 1.9274 - accuracy: 0.2592 - val_loss: 0.8328 - val_accuracy: 0.7018
Epoch 2/6
102/102 [==============================] - 77s 755ms/step - loss: 0.7174 - accuracy: 0.7444 - val_loss: 0.4143 - val_accuracy: 0.8527
Epoch 3/6
102/102 [==============================] - 77s 760ms/step - loss: 0.2891 - accuracy: 0.9041 - val_loss: 0.2049 - val_accuracy: 0.9294
Epoch 4/6
102/102 [==============================] - 77s 755ms/step - loss: 0.1519 - accuracy: 0.9503 - val_loss: 0.1049 - val_accuracy: 0.9608
Epoch 5/6
102/102 [==============================] - 77s 755ms/step - loss: 0.0732 - accuracy: 0.9751 - val_loss: 0.1196 - val_accuracy: 0.9616
Epoch 6/6
102/102 [==============================] - 77s 752ms/step - loss: 0.0810 - accuracy: 0.9771 - val_loss: 0.1206 - val_accuracy: 0.9599

model.save('1rst-model.h5')

Visualize the filters

from tensorflow.keras.preprocessing.image import img_to_array, load_img
img_path=test_df['Filepaths'][5221]
# Define a new Model, Input= image 
# Output= intermediate representations for all layers in the  
# previous model after the first.
successive_outputs = [layer.output for layer in model.layers[1:]]
#visualization_model = Model(img_input, successive_outputs)
visualization_model = tf.keras.models.Model(inputs = model.input, outputs = successive_outputs)
#Load the input image
img = load_img(img_path, target_size=(224, 224))
# Convert ht image to Array of dimension (150,150,3)
x   = img_to_array(img)                           
x   = x.reshape((1,) + x.shape)
# Rescale by 1/255
x /= 255.0
# Let's run input image through our vislauization network
# to obtain all intermediate representations for the image.
successive_feature_maps = visualization_model.predict(x)
# Retrieve are the names of the layers, so can have them as part of our plot
layer_names = [layer.name for layer in model.layers]
for layer_name, feature_map in zip(layer_names, successive_feature_maps):
  print(feature_map.shape)
  if len(feature_map.shape) == 4:
    
    # Plot Feature maps for the conv / maxpool layers, not the fully-connected layers
   
    n_features = feature_map.shape[-1]  # number of features in the feature map
    size       = feature_map.shape[ 1]  # feature map shape (1, size, size, n_features)
    
    # We will tile our images in this matrix
    display_grid = np.zeros((size, size * n_features))
    
    # Postprocess the feature to be visually palatable
    for i in range(n_features):
      x  = feature_map[0, :, :, i]
      x -= x.mean()
      x /= x.std ()
      x *=  64
      x += 128
      x  = np.clip(x, 0, 255).astype('uint8')
      # Tile each filter into a horizontal grid
      display_grid[:, i * size : (i + 1) * size] = x
# Display the grid
    scale = 20. / n_features
    plt.figure( figsize=(scale * n_features, scale) )
    plt.title ( layer_name )
    plt.grid  ( False )
    plt.imshow( display_grid, aspect='auto', cmap='viridis' )

(1, 111, 111, 64)
(1, 109, 109, 64)
(1, 54, 54, 64)
(1, 52, 52, 32)
(1, 26, 26, 32)
(1, 24, 24, 32)
(1, 12, 12, 32)
(1, 4608)
(1, 512)
(1, 128)
(1, 128)
(1, 128)
(1, 128)
(1, 9)

Performances

accuracy = history.history['accuracy']
val_accuracy  = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

plt.figure(figsize=(15,10))

plt.subplot(2, 2, 1)
plt.plot(accuracy, label = "Training accuracy")
plt.plot(val_accuracy, label="Validation accuracy")
plt.legend()
plt.title("Training vs validation accuracy")


plt.subplot(2,2,2)
plt.plot(loss, label = "Training loss")
plt.plot(val_loss, label="Validation loss")
plt.legend()
plt.title("Training vs validation loss")

plt.show()

pred = model.predict(test_images)
pred = np.argmax(pred, axis=1)

labels = train_images.class_indices
labels = dict((v,k) for k, v in labels.items())

print(labels)
print(pred)

{0: 'Black Sea Sprat', 1: 'Gilt-Head Bream', 2: 'Hourse Mackerel', 3: 'Red Mullet', 4: 'Red Sea Bream', 5: 'Sea Bass', 6: 'Shrimp', 7: 'Striped Red Mullet', 8: 'Trout'}
[0 0 6 ... 3 3 4]

y_pred = [labels[k] for k in pred]
print(classification_report(test_df.Labels, y_pred))
print('--------------------------------')
print(confusion_matrix(test_df.Labels, y_pred))
print('--------------------------------')

                    precision    recall  f1-score   support

   Black Sea Sprat       0.97      0.99      0.98       150
   Gilt-Head Bream       0.92      0.99      0.95       169
   Hourse Mackerel       0.98      0.99      0.98       135
        Red Mullet       0.99      0.99      0.99       156
     Red Sea Bream       0.99      0.92      0.95       159
          Sea Bass       0.97      0.93      0.95       149
            Shrimp       1.00      0.96      0.98       144
Striped Red Mullet       0.96      0.96      0.96       154
             Trout       0.93      0.96      0.94       134

          accuracy                           0.97      1350
         macro avg       0.97      0.97      0.97      1350
      weighted avg       0.97      0.97      0.97      1350

--------------------------------
[[149   0   0   0   0   1   0   0   0]
 [  0 167   0   0   1   1   0   0   0]
 [  0   1 133   0   1   0   0   0   0]
 [  0   0   0 155   0   0   0   1   0]
 [  0   4   0   0 147   0   0   0   8]
 [  3   4   1   1   0 139   0   0   1]
 [  1   0   0   0   0   0 138   5   0]
 [  1   0   2   1   0   2   0 148   0]
 [  0   5   0   0   0   1   0   0 128]]
--------------------------------

model.evaluate(test_images)[1]

22/22 [==============================] - 14s 617ms/step - loss: 0.1145 - accuracy: 0.9659

0.965925931930542